Collimator Summary
Link to Google Drive
Link to all documents regarding BSM2 model and wastewater treatment plant
Latest accounts and files
Collimator Accounts
- The main account is
hydrotrek@gqc.com. - Files are also shared with
pavan@gqc.com.
Projects
BSM2- The main project. This is shared with the following accounts.
BSM2 - backup - 3/21/23→ A backup of the aboveBSM2project created on 3/21/23 inhydrotrek@gqc.com
BSM2 - Copy - 2023-12-04 includes Brians changes to Vannary's original→ A copy in which Brian had made changes to the originalBSM2. Created on 12/04/2023.
Hydrotrek_AI→ An older copy of the main project which contains some hybrid modeling changes. The latest AI related changes are present as a separate file underBSM2project. See more details below.
Downloaded project files
Collimator does not allow downloading the model files. Only data files (
.csvin our case) and.pyfiles are downloadable.
Downloaded data files are uploaded to the gdrive location here.
Link to notebook related to Collimator project
- Collimator to TDEngine notebook
- To displace collimator's result and matlab's result on Grafana.
- matlab_output_cross_reference.csv (old filename: matlab_output_column_name.csv) contains the names of column in the matlab output file that are corresponding to the 16 variables of each submodel.
- collimator_cross_reference_2022_12_08.csv (old filename: collimator_output_column_name_xxxx_xx_xx) contains the names of column in the colimator output file that created on 2022-12-08 or after that are corresponding to the 16 variables of each submodel.
- The cross reference csv for different timestamp was created to match the column name of the 16 variables from the collimator output files that were created before 2022-12-08 because the blocks were renamed.
- Compare the physical model results with hybrid model (bioreactor_1 AI) results
Model descriptions inside BSM2 project
| Name | Type | Description |
|---|---|---|
| BSM2_Develop | Model | BSM2 developing model. The current working version is the BSM2 physical model that uses python script block for flow combiner and Bioreactor_with_Integrator_as_a_Delay_Block submodels are used for bioreactor 1, 2 ,3 ,4, and 5 in the activated sludge. |
| BSM2_with_Bioreact_1_AI | Model | BSM2 with activated sludge (AS) bioreactor 1 is replaced with a time series AI model. |
| BSM2_Physical_Model | Model | BSM2 model created based on BSM2 Matlab model. Bioreactor_with_Integrator_as_a_Delay_Block submodels were used for bioreactors 1, 2, 3, 4, and 5 in the activated sludge |
| outdatedBSM2_with_the_First_Bioreactor_Block(incorrect) | Model | BSM2 model with the old bioreactor submodels for bioreactor 1, 2, 3, 4, and 5 in the activated sludge. |
| Test_PyTorch_Model_Loading | Model | Testing pytorch model loading |
| Test_Delay_Block | Model | Testing the unit delay block by inputting different pulse width and period. |
| Test_Delay_Block_for_TS | Model | Testing the delay block for time series model |
| Test_Bioreactor_1_AI | Model | Testing time series bioreactor 1 AI model |
| Test_Mux_Block | Model | Testing the mux block |
| Test_Bioreactor_with_Integrator_as_a_Delay_Block_Submodel | Model | Testing the Bioreactor with Integrator as a delay block submodel |
| Test_Bioreactor_with_Integrator_as_a_Delay_Block_and_So_Control_Submodels | Model | Testing the Bioreactor with Integrator as a Delay Block and So Control submodels |
| Bioreactor_with_Unit_Delay_as_a_Delay_Block | Submodel | Physical model of the bioreactor in the activated sludge. The unit delay block is used to break the algebraic loop when reading the Kla value back from the So control submodel. |
| Bioreactor_with_Integrator_as_a_Delay_Block | Submodel | Physical model of the bioreactor in the activated sludge. The integrator block is used to break the algebraic loop when reading the Kla value back from the So control submodel. |
| Bioreactor_1_py | Submodel | Time series model trained with bioreactor 1 input and output values. |
| T_Model_Block | Submodel | Temperature model block. User can choose between using ODE to get the temperature output or setting the output temperature to be equal to the input temperature. Declare Tmodel value to be more than 0.5 to use the ODE to get the output temperature. |
| ODE_Mass_Balance_Block | Submodel | Generic ODE equation block for BSM2 variables. The block can be used to get soluble and particulate concentrations of different sub-processes. |
| TS_12_Timesteps | Submodel | Creating an input array for time series model. The array contains the current and the previous 11-time steps value. If the current time step is less than 11, the variable's initial value will be used instead. |
| So_Control | Submodel | Oxygen concentration control block |
Different Saved versions inside BSM2 project
- To access different saved version history of the model
- click on the version history tab on the upper right corner
- click on the version history tab on the upper right corner
| Version History | Model/Submodel | Description |
|---|---|---|
| pH_solver_ODE_for_7variables | BSM2_Develop | The simulation was aborted due to time out. Unlike the previous version, all 7 variables inside pH solver block were solved using ODE. |
| pH_solver_for_ODE_only_one_variable | BSM2_Develop | The simulation was able to be completed. However, the outputs were different from Matlab outputs. In attempt to work around algebraic loop found in BSM2_model_problem_with_AlgebraicLoop, I converted the Sh2_solver and pH_solver from DAE to ODE. In this version, only Sh2 and H_ion ( 1 out of 7 variables) in pH ODE were solved. The other 6 variables were solved based on the H_ion. |
| completed_model_old_SO4_control_block | BSM2_Develop | First version of the completed BSM2 model with an old SO4 control block for bioreactor 4 and the first version of the bioreactor submodel. |
| new_SO4_control_block_new_bioreactor_1 | BSM2_Develop | Modified bioreactor 1 block and SO4 control block. |
| replace_all_bioreactors_with_new_one | BSM2_Develop | Replaced bioreactor 1,2, 3, 4, and 5 with the modified bioreactors from the previous version (second version of bioreactor) |
| version_before_replacing_bioreactor_submodel | BSM2_Develop | bioreactor 1, 2, 3, 4, and 5 with modified bioreactor models using the old submodel block |
| version_after_replacing_new_bioreactor_submodels | BSM2_Develop | Replaced bioreactor 1, 2, and 3 with Bioreactor_with_Unit_Delay_as_a_Delay_Block submodel (third version of bioreactor) |
| bioreactor_without_delay_block_for_1,2,3,and5 | BSM2_Develop | Replaced bioreactor 1, 2, 3, and 5 with Bioreactor_without_delay_block. Only bioreactor 4 contains the unit delay |
| bioreactor_with_intergrator_delay | BSM2_Develop | Replaced bioreactor 1, 2, 3, 4, and 5 with Bioreactor_with_Integrator_as_a_Delay_Block submodels |
| bioreactor_with_intergrator_delay_new_flowcombiner | BSM2_Develop | Previous version with modified flowcombiner blocks |
| first_physical_model | BSM2_Develop | The first version of the physical model before replacing the bioreactor 1 with AI model |
| with_Bioreactor_1_AI | BSM2_Develop | Replaced bioreactor 1 block with Bioreactor_1_py submodel. The bioreactor_1_py block has a 'soluble-particulate-t-TSTPlus-01-11-23 version. |
| Physical model | BSM2_Develop | The most up to date version of the physical model |
| soluble-particulate-t-TSTPlus-01-11-23 | Bioreactor_1_py | Use this version to load soluble-particulate-t-TSTPlus-01-11-23.pkl model |
| soluble-particulate-t-inceptionTimePlus-01-11-23 | Bioreactor_1_py | Use this version to load soluble-particulate-t-inceptionTimePlus-01-11-23.pkl model |
Input files
- We use the dryinfluent dataset from MATLAB as our input to the model. They are the same input dataset to the MATLAB model.
- The data has the time step of 15 min (0.0104 day).
Initialization Script
- We can import py script to install certain python package to be used in python script block before starting the simulation runs. For an example, we need to
pip install ipython tsai torch ipykernelbefore running BSM2_with_Bioreactor_1_AI model.
Description of csv files
| no | csv file | description | location |
|---|---|---|---|
| 1 | influent_Q_temp.csv* | model's input flow and temperature. The data has a time step of 15 min (0.0104 days). The data is from MATLAB's BSM2 model dryinfluent dataset. | Input of all types of BSM2 models |
| 2 | influent_time_SI_SS_XI.csv* | model's input Si and Ss and Xi. The data has a time step of 15 min (0.0104 days). The data is from MATLAB's BSM2 model dryinfluent dataset. | Input of all types of BSM2 models |
| 3 | influent_SO_SNO_SNH_SND.csv* | model's input So, Sno, Snh, and Snh. The data has a time step of 15 min (0.0104 days). The data is from MATLAB's BSM2 model dryinfluent dataset. | Input of all types of BSM2 models |
| 4 | influent_XND_SALK_TSS.csv* | model's input Xnd, Salk, and TSS. The data has a time step of 15 min (0.0104 days). The data is from MATLAB's BSM2 model dryinfluent dataset. | Input of all types of BSM2 models |
| 5 | influent_XS_XBH_XBA_XP.csv* | model's input Xs, Xbh, Xba, and Xp. The data has a time step of 15 min (0.0104 days). The data is from MATLAB's BSM2 model dryinfluent dataset. | Input of all types of BSM2 models |
| 6 | Particulate2TSS.csv | the ratio of Xi, Xs, Xbh, Xba, and Xp convert to TSS at the end of the primary clarifier, bioreactors, activated sludge, thickener, and dewatering. | uses in the python script block inside the particulate block of each process |
| 7 | INTERFACEPAR.csv | parameters used to convert activated sludge state variables to anaerobic digester state variables. | found in ASM2ADM and ADM2ASM python script blocks inside an AnaerobicDigester/ |
| 8 | DIGESTERPAR.csv | parameters used to calculate the reaction rate of the anaerobic digester state variables. | found in reac, pH_solver, and mapping_output python script blocks inside AnaerobicDigester/ADM1_1 |
| 9 | DIGESTERPARINIT.csv | initial state of anaerobic digester state variables | uses in pH_solver inside AnaerobicDigester/ADM1_1 |
| 10 | Qbypass_split_param.csv | contain three types of bypass strategies and their control flow values. | uses in Q_bypass_Plant_split python script block |
| 11 | BIOREACTORPAR.csv | parameters used to calculate the reaction rate of the state variables. | uses in reac python script block inside all 3 types of bioreactor submodel |
| 12 | AS_in_Q_T.csv | Activated sludge influent flow and temperature values from MATLAB model. The values are the inputs to the test bioreactor 1 submodel | input of all types of test_Bioreactor submodel to simulate the run for bioreactor 1 |
| 13 | AS_in_soluble.csv | Activated sludge influent soluble state variables values from MATLAB model. The values are the inputs to the test bioreactor 1 submodel | input of all types of test_Bioreactor submodel to simulate the run for bioreactor 1 |
| 14 | AS_in_particulate.csv | Activated sludge influent particulate state variables values from MATLAB model. The values are the inputs to the test bioreactor 1 submodel | input of all types of test_Bioreactor submodel to simulate the run for bioreactor 1 |
| 15 | reac3_Q_T.csv | Bioreactor 3 influent flow and temperature values from MATLAB model. The values are the inputs to the test bioreactor 3 submodel | input of all types of test_Bioreactor submodel to simulate the run for bioreactor 3 |
| 16 | reac3_soluble.csv | Bioreactor 3 influent soluble state variables values from MATLAB model. The values are the inputs to the test bioreactor 3 submodel | input of all types of test_Bioreactor submodel to simulate the run for bioreactor 3 |
| 17 | reac3_particulate.csv | Bioreactor 3 influent particulate state variables values from MATLAB model. The values are the inputs to the test bioreactor 3 submodel | input of all types of test_Bioreactor submodel to simulate the run for bioreactor 3 |
| 18 | reac1_Q_T.csv | Bioreactor 1 influent flow and temperature values from MATLAB model. The values are the inputs to the test bioreactor 1 submodel. These values are the outputs of the hydraulic delay block. | input of all types of test_Bioreactor submodel to simulate the run for bioreactor 1 |
| 19 | reac1_soluble.csv | Bioreactor 1 influent soluble state variables values from MATLAB model. The values are the inputs to the test bioreactor 1 submodel. These values are the outputs of the hydraulic delay block. | input of all types of test_Bioreactor submodel to simulate the run for bioreactor 1 |
| 20 | reac1_particulate.csv | Bioreactor 1 influent particulate state variables values from MATLAB model. The values are the inputs to the test bioreactor 1 submodel. These values are the outputs of the hydraulic delay block. | input of all types of test_Bioreactor submodel to simulate the run for bioreactor 1 |
| 21 | test_delay_time_series_input.csv | input data to test test_Delay_Block_for_TS submodel. | input of test_Delay_Block_for_TS |
* These csv can be combined into one file. The input csv was splitted into 5 different files because initially the dataloader could not read csv with more than four columns.
Description of py and pkl files
| no | py/pkl files | description | location |
|---|---|---|---|
| 1 | intialization.py | select this file under `INITIALIZATION SCRIPT` to 'pip tsai' before the simulation. | INITIALIZATION SCRIPT in the properties tab |
| 2 | install_python_mods(1).py | select this file under `INITIALIZATION SCRIPT` to 'pip install ipython tsai torch' before the simulation. | INITIALIZATION SCRIPT in the properties tab |
| 3 | install_python_mods.py | select this file under `INITIALIZATION SCRIPT` to 'pip install ipython tsai torch ipykernel' before the simulation. | INITIALIZATION SCRIPT in the properties tab |
| 4 | soluble-particulate-t.pkl | outdated time series models of Bioreactor 1 submodel. | not used in any model |
| 5 | soluble-particulate-t-InceptionTimePlus-01-11-23.pkl | time series models of Biorector 1 submodel to predict the next state variable values using the values from the previous 12 time steps | Choose soluble-particulate-t-InceptionTimePlus-01-11-23 Version of BIOREACTOR_1_PY located inside ActivatedSluge to use this model before starting the simulation |
| 6 | soluble-particulate-t-TSTPlus-01-11-23.pkl | time series models of Biorector 1 submodel to predict the next state variable values using the values from the previous 12 time steps | Choose soluble-particulate-t-TSTPlus-01-11-23 Version of BIOREACTOR_1_PY located inside ActivatedSluge to use this model before starting the simulation |